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As requested in your letter of 4/4/18, the following provides additional information about the 
development of tools supported by this grant. This response is divided into two parts to 
correspond to the two tools: ClientView and the Allegheny Family Screening Tool. 

CLIENTVIEW 

The first phase of the project, developing the foundation of the application, was successfully 
completed in January 2016 with a focus on improving the user experience and meeting current 
user expectations for internet-based applications. Additional information sources were also 
added including DHS's common assessments, the Child Adolescent Needs and Strengths 
Assessment (CANS) and the Family Assessment of Strengths (FAST), as well as additional client 
documents stored in our countywide document management system. These documents 
include child welfare safety assessments, dependency petition hearing summaries, petitions for 
dependency, risk assessments, shelter care applications and transition plans. During this initial 
phase, the capacity to track and display the names of client workers currently and formerly 
working with this individual was added to Client View to support case coordination. 

The second phase of the project, expanding access to providers, began in February 2016 with a 
pilot group of providers and ended in May, at which point, DHS began a phased rollout of Client 
View within each DHS service system, providing in-person training and guidance on the use of 
the system. 

To date. Client View has been rolled out and made available to a majority of DHS contracted 
providers representing the array of service systems administered by DHS. Client View has been 
well-received by the Provider Agencies, and we have seen a lot of enthusiasm surrounding the 
use of the system and being able to incorporate this new tool into their business processes. 
Provider staff have been trained on giving access to clients so that they can initiate a 
conversation together about their history and use the information to strategize next steps and 
to provide the best care possible. 


1 



The third and final phase of the project, focused outreach to clients, has experienced some 
delays, but is nearing large scale rollout. Because Client View is developed upon a data 
warehouse model, giving clients access to Client View is intrinsically more difficult than what 
health care providers experience in sharing an electronic health record. In an electronic health 
record model, the record is established, and information is downloaded into it. In a data 
warehouse model, distinct source system records that already exist are being matched to one 
another to determine shared identities, and a record is established based on this matching 
process. This model has proven effective for the majority of client records but was not 
structured appropriately to address exceptional cases where a client record was duplicated or 
incorrectly matched. For example, in some service systems such as homelessness, the program 
may not be able to collect enough information to provide a credible match and our ability as a 
system administrator to link these files was limited. In response, DHS had to take a step back 
and initiate a significant change to the identity tables within the warehouse. This change now 
allows us to make and retain manual data changes to address incorrect matches and/or 
duplicated records. 

We are now testing and refining this functionality. Once completed, we will begin doing 
targeted outreach to encourage clients to establish an account and use Client View to manage 
and coordinate their own care. 

A key reason for the delay in the client-facing phase of the project is that there has been no 
easy answer to the question: Where can clients go to access Client View? Right now, the only 
place they could access it is on www.AllefihenvCountvAnalvtics.US, which is not client friendly 
or intuitive. We need a place that makes better sense for clients. In response, DHS issued and 
recently closed an RFP for a Client Experience Website, to serve as a digital front door for DHS 
clients and Allegheny County residents to access tools (including Client View) and services. 

The RFP closed on March 7^^ with 18 proposals received. Currently, the evaluation team is 
reviewing and evaluating proposals; the team will meet on April 4^^*^ to rank proposals and select 
those proposers who will be invited to make an oral presentation. We anticipate that successful 
proposers will be notified in June, at which time the build will begin. 

Use of data, client outcomes and allocation of resources 

As we have seen over nearly two decades with our own implementation of Client View (and its 
predecessor Data View), the uptake and full operational benefits of this system takes time to 
achieve. But even now, we see the system assisting DHS and providers to more effectively 
allocate resources. 

Clients rarely can remember and share their entire service history during intake. Now provider 
staff no longer rely exclusively on an Individual's or family's collective memory. This leads to 
reduced initial and follow-up time spent discussing history and more time available for service 
planning and coordination. Additionally, the seamless integration of Client View within our 
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transactional source systems eliminates additional steps for the care coordinator who can 
access a client's cross system record within Client View by clicking a link provided directly 
within that system (without having to login again). 

Before they begin providing referrals or developing service plans, provider staff are now better 
informed about other services a client may be receiving and have contact information to reach 
out to clinicians and care providers at those agencies. Knowing the totality of a client's service 
history and potential needs allows care coordinators to make better service choices. For 
example, a homelessness program may recommend a permanent supportive housing option 
instead of rental assistance for an individual with a history of mental illness. 

This reduces the number of housing and/or service-related placements that fail because of 
Incomplete service histories. Client view also supports simple but potentially time-consuming 
and costly daily activities. For example, staff needing more information about a client to 
complete a benefit eligibility application can find things like race, address, social security 
number and date of birth in the system to expedite those tasks, where the alternative would be 
to call, visit or in other ways request the information from the client which would potentially 
delay the actual application for and receipt of the benefit. 

This data in Client View was previously unavailable to our contracted providers, so for the first 
time they are now able to make full use of it, improving their ability to assess the quality of 
their programs. With the support of this application, they can now do activities such as case 
reviews and root cause analysis to understand at a greater breadth the circumstance of their 
client's needs and strengths, and to determine whether the supports they are providing are 
appropriate for the population they are serving. 

For the clients, this information sharing now and into the future means better historical 
documentation of their care for themselves, improved coordination of their care by DFIS and 
provider staff, less time spent repeating the same demographic and service history information, 
fewer referrals for duplicative services, and fewer failed service arrangements. Ultimately, 
providing accurate and complete information about a client empowers care providers in 
partnership with those they serve, to make informed and appropriate service decisions that will 
better benefit the client. 

Staff, provider and consumer satisfaction survey results 

In February and March of 2019, the Allegheny County Department of Fluman Services (DHS) 
surveyed its employee base and provider network to request feedback about the user 
experience for the Client View system. The purpose of the survey was to gain insight into how 
often users log into the system and the level of satisfaction with the system overall. Users were 
given the opportunity to comment and make suggestions as well. Out of 2029 surveys, we 
received 170 anonymous responses (133 DFIS Employees and 37 Provider users). This report 
summarizes those results. 
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Frequency of Use: 






Simplicity of Use: 

Users were asked to indicate how easy or difficult the system is for them to use. Seventy-two 
percent of users indicated that the system was either extremely or somewhat easy. 

Q5 - How easy or difficult is it to use Client View? 


28% 

44% 

24% 

Ext Jomety easy 

SnnuTwIi.it 

Mrither I'oi fliffimilt 


0 Extremely easy (28%) 9 Somewhat easy (44%) ■ Neither easy nor difficult (24%) fl Somewhat difficult (3%) 

■ Extremely difficult (1%) 

n=170 



DHS Employee 

Provider 


% 

Count 

% 

Count 

Extremely easy 

30% 

40 

22% 

8 

Somewhat easy 

44% 

59 

41% 

15 

Neither easy nor 
difficult 

22% 

29 

32% 

12 

Somewhat difficult 

2% 

3 

5% 

2 

Extremely difficult 

2% 

2 

0% 

0 

Total 


133 


37 


Overall Satisfaction: 

Users were asked to indicate a level of overall satisfaction with Client View. Sixty-eight percent 
of users indicated that they were either extremely or somewhat satisfied. Nearly all of the users 
that replied that they were neither satisfied nor dissatisfied indicated that they never or rarely 
use the system. Users were asked to explain why they chose the response they did, and many 
users that replied 'dissatisfied' indicated that the page loading times could be improved. Some 
users also expressed frustration with duplication and lag time with data refreshes. Because of 
this feedback, DHS is already working to optimize performance and decrease loading times for 
users. 


5 
















Q6 - Overall, how satisfied or dissatisfied are you with the system? 


19% 

lix'tromoly sntisfircl 

49% 

Somewhat 5?ati3ficd 

24% 

Neither satiatied nor di^satistied 


■ Extremely satisfied (19%) 

■ Somewhat satisfied (49%) ■ Neither satisfied nor dissatisfied (24%) ■ Somewhat dissatisfied (6%) 


■ Extremely dieutisfled (1%) 


n=170 



DHS Employee 

Provider 


% 

Count 

% 

Count 

Extremely satisfied 

19% 

25 

19% 

7 

Somewhat 

satisfied 

50% 

67 

43% 

16 

Neither satisfied 
nor dissatisfied 

23% 

30 

27% 

10 

Somewhat 

dissatisfied 

8% 

11 

8% 

3 

Extremely 

dissatisfied 

0% 

0 

3% 

1 

Total 


133 


37 


Availability of Information: 

Users were asked about the information that is available in the system and whether they were 
satisfied or dissatisfied with the datasets currently available. In this case, 96 percent of users 
were satisfied or responded neutrally regarding the information that is available to them in 
Client View. Respondents were also given the opportunity to suggest improvements to the data 
or additional information that would be beneficial to end users. 
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Q9 - Overall, how satisfied or dissatisfied are you with the datasets 
available in system? 


19% 45% 32% 

hxtrcmely Somc-wliot satisfied Neither s.nti3fied nor dissotisfied 


H Extremely satisfied (19%) 0 Somewhat satisfied (45%) 0 Neither satisfied nor dissatisfied (32%) 

0 Somewhat dissatisfied (3%) 0 Extremely dissatisfied (1%) 

n=170 
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4 
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60 
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dissatisfied 

3% 

4 

3% 

1 

Extremely 

dissatisfied 

0% 

0 

3% 

1 

Total 


133 


37 


Next Steps: 

The results of this survey are being used to improve system functionality and will help to steer 
future enhancements and training. Some respondents raised questions that will be addressed 
in a series of short 'helpful hint' emails highlighting features and functions that could benefit all 
users. DHS values the time and consideration of all respondents and appreciates the comments 
and suggestions received. We plan to conduct another survey in the future to assess how we 
have improved. 
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ALLEGHENY FAMILY SCREENING TOOL (AFST) 

Since the August 2016 implementation of the Allegheny Family Screening Tool (AFST), a number 
of changes have been made in response to ongoing monitoring and user feedback. These 
changes, which include the predictor fields used, the modeling method, and policies about how 
the tool is used, are described in an updated methodology report which we expect to publish by 
the end of April. In addition, we have received the first draft of the impact evaluation, which we 
also intend to publish within the next few weeks along with a summary intended for a more 
general audience and frequently-asked questions. The information below is based on findings 
from the impact evaluation. 

Flow is the tool impacting improved client outcomes? 

The AFST was designed to improve screening decisions, which we believe will lead to better 
safety outcomes for children as well as better outcomes for families. 

Evaluation findings are detailed below, and emerge from a set of methodologically strong, 
quasi-experimental methods (i.e., interrupted time series analyses, generalized linear models, 
and difference-in-difference techniques). The evaluation primarily consists of outcome 
comparisons for the approximately 31,000 children who were reported for alleged 
maltreatment during the 18-month period before the AFST was implemented (Pre-AFST: 

January 1, 2015 through July 31, 2016) to those observed for the 34,000 children reported after 
the AFST was fully implemented (Post-AFST: December 1, 2016 through May 31, 2018). 
Outcomes for both groups (Pre-AFST vs. Post-AFST) were examined for 15 to 17 months after 
the initial maltreatment report was received. 

Key findings include: 

1. Implementation of the AFST increased the accuracy of maltreatment referral 
screening decisions for higher risk children. Use of the tool led to an increase in 
screen-in rates for higher-risk children who needed intervention supports. Specifically, 
there was a statistically significant increase in the proportion of children screened-in 
who then had a child welfare case opened or, if no case was opened, were re-referred 
within 60 days. This effect appears to have partially attenuated over time. 

2. The AFST led to reductions in screening disparities between black and white children 
with similar risk levels. Use of the tool led to a statistically significant reduction in the 
proportion of black children who had cases open and a proportionate increase in white 
children. This meant that the pre-AFST disparity in case opening rates was eliminated. 
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3. Overall, the AFST did not lead to increases in the number of children screened-in for 
investigation. Use of the tool appears to have resulted in a different pool of children 
screened-in for investigation - including more children who needed intervention 
supports. But from a workload perspective, there was no significant increase in the 
number or proportion of children investigated among all children reported for 
maltreatment. 

4. There was no evidence that the AFST resulted in greater screening consistency within 
individual call-screeners. Specifically, for the subgroup of 11 call screeners who 
handled a substantial volume of both Pre-AFST and Post-AFST referrals, attempts were 
made to assess whether the AFST led to more "within-screener" consistency. Likewise, 
changes in screening consistency by age group and racial group were also assessed. No 
changes were detected, although it should be noted that there was likely insufficient 
power to identify anything other than very large shifts. 

How is the tool impacting better use of data? 

The AFST represents the next generation of data-based decision making at DHS. It is the first 
predictive analytics tool developed to more comprehensively and effectively use data from the 
Data Warehouse to make screening decisions about child welfare referrals. 

How is the tool impacting more effective allocation of resources? 

Increased accuracy in screening decisions (a positive outcome of the AFST) has led to more 
effective allocation of investigation resources (i.e., more resources are being spent to 
investigate the higher-risk cases). 


Audited financial statements 


Available at http://www.alleRhenvcontroller.cQm/audit/13/annual-reports/report/comprehen5ive 



Marc Cherna, DHS Director 
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